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We investigate dynamics of recurrent neural networks with correlated noise to analyze the 
noise's effect. The mechanism of correlated firing has been analyzed in various models, but 
its functional roles have not been discussed in sufficient detail. Aoyagi and Aoki have shown 
that the state transition of a network is invoked by synchronous spikes. We introduce two 
types of noise to each neuron: thermal independent noise and correlated noise. Due to the 
effects of correlated noise, the correlation between neural inputs cannot be ignored, so the 
behavior of the network has sample dependence. We discuss two types of associative memory 
models: one with auto- and weak cross-correlation connections and one with hierarchically 
correlated patterns. The former is similar in structure to Aoyagi and Aoki's model. We show 
that stochastic transition can be presented by correlated rather than thermal noise. In the 
latter, we show stochastic transition from a memory state to a mixture state using correlated 
noise. To analyze the stochastic transitions, we derive a macroscopic dynamic description as 
a recurrence relation form of a probability density function when the correlated noise exists. 
Computer simulations agree with theoretical results. 

KEYWORDS: common external input, correlated noise, associative memory, stochastic transi- 
tion, macroscopic dynamic description 



1. Introduction 

In the activities of nerve cells, synfire chains, i.e., synchronous firings of neurons, can often 
be observed. 1 To analyze the mechanism of synchronous firings, conditions for propagating 
them between layers have been investigated in layered neural networks. 2 ' 3 Common synaptic 
inputs to neurons have been introduced in layered neural networks, and correlated firings 
of neurons have been investigated on a theoretical level. 4 We have also analyzed common 
synaptic inputs in recurrent neural networks, i.e., a sequential associative memory model, 5 
and succeeded in deriving a macroscopic dynamic description as a recurrence relation form 
of a probability density function. On another front, investigating the functional roles of syn- 
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chronous firings has become important. Associative memory models, which store patterns as 
attractors, are used to explain these roles. Aoyagi and Aoki 6 ' have investigated a model with 
auto- and weak cross-correlation connections and showed that a transition between attractors 
cannot be invoked by thermal independent noise but can be by synchronous spikes. In general, 
when using thermal independent noise, the state leaves an attractor for another one. How- 
ever, Aoyagi and Aoki's model can make the state transit to the next attractor by synchronous 
spikes, i.e., external inputs. 

In this paper, we consider two types of associative memory models with a common ex- 
ternal input. The common external input affects all neurons equally and therefore invokes 
correlated firings of neurons. To analyze dynamic behavior theoretically, we assume that com- 
mon external input obeys Gaussian distribution and acts like correlated, not independent, 
noises. 

The first model consists of an autoassociative memory model 8 ' 9 and a weakly connected 
sequential associative memory model, 10-13 as well as Aoyagi and Aoki's model. 6,7 Associa- 
tive processing requires functions of both autoassociative memory, which retrieves a memory 
pattern from an ambiguous initial state, and sequential associative memory, which retrieves 
memory patterns episodically. 14 This model contains these functions. To switch from autoas- 
sociative to sequential associative, we introduce external inputs. One input is an independent 
(thermal) noise and the other is a correlated noise (a common external input). The syn- 
chronous spikes in Aoyagi and Aoki's model can be considered correlated noises in our model 
since they invoke correlated firings. We examine the effectiveness of the independent and cor- 
related noises, and demonstrate that stochastic transition can be presented by correlated, but 
not independent, noise, as shown in Figure 1(a). 

The second model we consider is an associative memory model that stores hierarchically 
correlated patterns. 15-19 In this case, both the memory states and their mixture states are 
attractors. 20 We demonstrate that a stochastic transition from a memory to a mixture state 
can be invoked by the correlated rather than the independent noise, as shown in Figure 1(b). 

Almost all existing models have been analyzed by applying independent units or neurons at 
the thermodynamic limit. Since we introduce correlated noises in these models, the sum of the 
inputs to the neurons are correlated. Therefore, the firings of the neurons are also correlated. 
With an infinite number of neurons, there is no thermodynamic limit and sample dependence 
appears. 4 ' 21 Theoretical approaches that address sample dependence are described by prob- 
ability density functions (PDF) for macroscopic states or order parameters. 5 ' 22 Yamana and 
Okada 22 have introduced uniform common synaptic inputs that depend on preneurons to a 
layered associative memory model and derived the PDF. Kawamura et a/. 5 have introduced 
common synaptic inputs to recurrent neural networks and derived a macroscopic dynamic de- 
scription as a recurrence relation form of the PDF. The common synaptic inputs that depend 
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Fig. 1. Schematic illustrations of (a) stochastic transition by correlated noise in an associative mem- 
ory model with auto- and weak cross-correlation connections, and (b) transition from memory to 
mixture state in one with hierarchically correlated patterns. 

on preneurons were analyzed in these previous papers. 23-25 In this paper, we analyze external 
rather than synaptic inputs and derive a macroscopic dynamic description as a recurrence 
relation form of the PDF. 

This paper is organized as follows. In the next section, we examine the first model, i.e., 
the model with auto- and weak cross-correlation connections. Theoretical results and those 
obtained using computer simulations are given. In §3, we examine the second model, i.e., the 
model that stores hierarchically correlated patterns. The final section is devoted to conclusion. 

2. Autoassociative and sequential associative memory model 

2.1 Model 

We consider the associative memory model consisting of N units or neurons and intro- 
duce two types of external input: independent external input £*, which affects each neuron 
independently, and common external input rf, which affects all neurons mutually. The state 
of the neurons takes x\ = ±1 and is updated synchronously by 

x^^F^pJ^ + Cj + v^j , (1) 

where the output function is F{h) = sgn(/i), and Jij is a synaptic connection from the j'th to 
the ith. neuron. Here, the synapse has both auto- and weak cross-correlation connections and 
stores p random patterns ^ = • • • , , n = 1, 2, ■ ■ ■ ,p. The element of the patterns 
takes ±1 with 

Prob[^f = ±1} = \. (2) 
The synaptic connection Jy is given by 

^• = 4E^ + ^E^, (3) 



N ^— ' 3 N 



where £f +1 = Cj an d £ denotes a connecting parameter. Under normal conditions, that is, 
without noise, the model behaves as an autoassociative memory model, since the connecting 
parameter is very weak e<l. 
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We define the overlap by the direction cosine between the state of neurons, a;*, at time t 



and the memory pattern £ M , 

N 

u 

m 



? = ^EeT4 (4) 

i=l 

From (3) and (4), the state of the neurons, x* +1 , becomes 



f ( E *M + £ E *T V + c* + ^ ) (5) 

K H=1 n=l 



= F \J2{^+^ +1 )< + ci+^j- (6) 

The initial state x° is determined according to the probability distribution 

Prob[z° = ±1] = (7) 

Therefore, the overlap between the pattern and the initial state x° is rriQ. 
2.2 Theory 

Let us derive the macroscopic description of the model with the external inputs. In order to 
simplify the argument, we assume that the number of memory patterns, p, is finite, especially 
when p = 3. The independent external input £| is time independent and independent of 
each neuron x\. In addition, we assume £| obeys the Gaussian distribution with A/" (0, A 2 ). 
The correlated external input rf is time independent, and is assumed to obey the Gaussian 
distribution with J\f (0, <5 2 ). These external inputs are called independent noise and correlated 
noise. 

2.2.1 Without correlated noise (rf = 0) 

First, we discuss the case without correlated noise, rf = 0. Since there is only independent 
noise, when N — > oo, the overlap m^ +1 , /x = 1, 2, 3 becomes 

<i = ^>:^i>:(c+^r i x+cn, w 



(9) 



^E^ fE(c+^ +1 x+cA, 

where D z = -J= exp ( — t) an ^ denotes the average over the memory pattern The 
function erf(it) is defined as 

erf (u) = 4= / <ftexp (-t 2 ) . (11) 
In this case, we find that the macroscopic parameter is deterministically given. 
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2.2.2 With correlated noise (rf ^0) 

In the case with correlated noise, the states of the neurons are correlated, because the 
common external input is introduced into all neurons mutually. Therefore, sample dependence 
must be taken into account. We derive a macroscopic dynamic description as a recurrence 
relation form of a probability density function (PDF). 

The correlated noise distributes rf ~ J\f(0,5 2 ). When rf is known at given time t and 
N — > oo, then m^ +1 , /x = 1, 2, 3 can be given as a function of m\ , m 2 , mf, and -rf , 

mf +1 K,m?,m?,,<) = // D Z ("F (£" + m? + Az + j \ , (12) 

_ ^( ^^X'"' 1 " "' ))/ <13> 
Using this equation, the dynamic behavior of the model for various 5 can be analyzed. When 
5 = 0, the model corresponds to the existing associative memory models and behaves de- 
terministically. However, when 5 > 0, the values of mj, m 2 , and are distributed. This 
distribution can be described as the PDF, p (m\ , m 2 , mf , rf) . Since rf is independent of raj, 
m 2 , and mf, p (m^, , mf , ?7*) can be divided into two PDFs: 

P { m h m u m t, rf) = P { m l,m 2 ,mf) P (V) ■ ( 14 ) 
Therefore, the PDF can be given by 



p(m t 1 +1 ,mf +1 ,m t 3 +1 ) = / dm v t drfp (mj, mf, mf) p (r?*) 

3 

H <5 (m^ +1 - mj , +1 (m t 1 ,m?,m?,»7 t )) , (15) 



3 

X 

where 5(-) denotes the Dirac delta function defined as 



0, u^O 



The PDF p (rf ) is given by 



2- 



^' ) = 7SI exp [-^l (17) 



Then, we combine this with the integral of rf and obtain equations 

3 



P , m 2 +1 , ml +1 ) = j[\ dm u t p (m\ , m 2 t , mf) If (m^ +1 ; mj') , (18) 

K[m v t+x -mX) = / dij'p^) n*Ki - <iK 1 . ff H J . fn t-'/)) ■ ( 19 ) 

i/=i 

We can also calculate (19) directly, but we evaluate it using the Monte Carlo method in the 
following section. 
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Fig. 2. Time evolutions of overlaps m\ , mf, and mf using a computer simulation in the case without 
correlated noise, where s = 0.1 and A = 0.6. 



2.3 Results 

The first case is without correlated noise if, that is, 5 = 0.0. Figure 2 shows time evolutions 
of the overlaps m\, mf , and mf using a computer simulation, where the standard deviation 
(SD) of the independent noise (* is A = 0.6. The connecting parameter is e = 0.1. Numbers 
in Figure 2 denote indexes of the memory patterns. We find that no transition occurs even 
if large independent noise is induced. The model acts like the autoassociative memory model 
through to the end. 

Next, we consider the case with correlated noise rf . Figure 3 shows time evolutions of 
the overlaps m\, mf, and mf using computer simulations (N = 60,000), where SDs of the 
independent and correlated noises are A = 0.1 and 5 = 0.37. Figure 3(a) and (b) show 
results of the different trials. Therefore, we can verify that sample dependence exists and 
that the correlated noise invokes stochastic transition. The model acts like the sequential 
associative memory model. In the case of smaller correlated noise, transition probability to 
the next attractor is much smaller. However, in the case of larger noise, the state becomes 
much noisier, and we cannot identify what pattern is retrieved. Therefore, the appropriately 
sized correlated noise should be introduced in the model. 

Let us consider sample dependence. Figure 4 shows the time evolution of the overlaps m\ , 
mf, and mf . The number of neurons is N = 60, 000. Each figure shows 20 samples. The initial 
overlap is mo = 1.0. When sample dependence appears, we need to consider the distribution, 
not the overlaps, of the samples. From the joint probability density function p (m|,mf,mf) 
in (18), we evaluate the marginal probability density function of mf , \i = 1, 2, 3, 

Figure 5 shows the marginal PDF of the overlaps, p (mj) ,p (mf) ,p (mf) at time t = 10, 
50 in Figure 4. Abscissas denote overlap mf, and ordinates denote marginal PDF p(mf) 
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Fig. 3. Time evolutions of overlaps m\,m^, and using computer simulations in the case with 
correlated noise, where e = 0.1, A = 0.6, and 5 = 0.37. (a) and (b) show results of different trials. 



on a logarithmic scale. Boxes denote histograms of 1,000 samples, which are obtained using 
computer simulations (TV = 60,000), and lines denote theoretical results. The theoretical 
results agree with the computer simulations. The state at time t = 10 distributes around the 
memory state since the overlap m\ takes the maximum in m\ = 1.0. The state at time 
t = 50 also distributes around that of £ 2 . As time passes, these PDFs approach a uniform 
distribution asymptotically. 

We discussed the case when a common external input obeys the Gaussian distribution. 
We can, however, choose alternative common external input. The state transition seems to be 
caused by certain common external input. We empirically found a particular common external 
input. Figure 6 shows the state transition using the following external input: 

1 , t mod 10 = 
0.5 , t mod 10 = 1 • (21) 
, otherwise 

Thick vertical lines denote the particular common external input rf. For instance, if a sequence 
of the external inputs is r/ 10 = 1 and rf 1 = 0.5, then the state transition £ 2 — > £ 3 can occur 
the next time. When this is repeated, the memory patterns are retrieved sequentially. In this 
manner, the common external input to the neurons can control the transitions. 

3. Associative memory model with hierarchically correlated patterns 

3.1 Model 

Now we discuss the associative memory model with hierarchically correlated patterns. 
Parent patterns fi = 1, 2 ■ ■ • ,p are generated at random, and then child patterns , v = 
1, 2, • • • , k are generated from the parent patterns £f with similarity r, that is, they are given 



rf 
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Fig. 4. 20 samples of overlaps ml, mf, and m\ starting from m = 1.0 using computer simulations, 
where e = 0.1, A = 0.1, and <5 = 0.37. (a) shows results of overlap ml, (b) for ml, and (c) for mf . 



by 



P[g = ±l] = 

p[tr=±i] = 



i 

2' 

1 + r£f 



(22) 



(23) 



The child patterns having different parents are independent of each other, but those having 
the same parent are correlated. In the limit of N — > oo, the correlations become 



E 



E 



, fi ^ p, 



1 , V = V 

„2 



r , v / v 

The synaptic connection stores only the child patterns (% ,v and is given by 

V k 



T — — V V cWtW 



(24) 
(25) 

(26) 
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Mixture states of the child patterns also become attractors even if only the child patterns are 
stored. 20 

To make the argument simple, we assume that the numbers of the parent and child pat- 
terns, p and k, are finite, especially when p = 1 and k = 3. When the similarity is r = 0, that 
is, the child patterns are independent of each other, the mixture states become unstable faster 
than the memory states with an increase in temperature. 26 When r > 0, the mixture states 
become stabler. We define the overlap by the direction cosine between the state of neurons, 
x l , at time t and the child pattern 

1 N 

< = ^£^4 ( 27 ) 

1=1 

From Equations (1), (26), and (27), the state x* +1 becomes 

F^l^ + Ct + ^y (28) 



3.2 Theory 

As in the case of the previous model, let us derive the PDF of this model. In the case 
without correlated noise rf, when N — > oo, the overlap m^ +1 can be 

3 



m t+i 



J D ^ F (E + Az J I - ( 29 ) 



where (-)^ denotes the average over not only the child patterns £ 1,K , but also their parent 
pattern £ 1 . 

In the case that rf would be known at given time t and N — ► oo, then m^ +1 could be given 
as the function of m], m 2 , mf , and rf , 



m^ +l (m l tJ m 2 t ,rnl,rf) 



^D^F^e^mt + Az + r,^ , (31) 

(^ rf (S^^)). 5 (32) 



Substituting Equations (18) and (19), we can obtain the recurrence relation form of the PDF 
for this model. 

3.3 Function of External Noise 

3.3.1 Without correlated noise (rf = 0) 

We show dynamic behaviors without correlated noise rf. To verify retrieval of a memory 
state of a child pattern, we evaluated retrieval processes for various initial overlaps. Figure 7 
shows the retrieval processes for the initial overlaps mo = 0.1, 0.2, • • • , 1.0. The abscissas and 
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ordinates denote the overlaps and mf. Solid and broken lines denote results obtained 
using computer simulations (N = 60,000) and Equation (10). When A < 0.3, the state is 
attracted to the attractor of for a significantly large initial overlap. When A = 0.4 or has 
a small initial overlap, the state goes to mixture state, not the memory state. We find that 
independent noise cannot invoke a transition to the mixture state once the state is attracted 
to the memory state. 

3.3.2 With correlated noise (rf ^ 0) 

We show dynamic behavior with correlated noise rf using computer simulations. In the 
case of 5 > 0, sample dependence appears and the network retrieves either memory or non- 
memory state even if retrieval from the same initial state. To examine functional roles of 
the correlated noise, we choose a sufficiently small independent noise, A = 0.2. Figure 8 
shows time evolutions of the overlap with correlated noise. The results are obtained from 20 
samples using computer simulations (N = 60,000), where 5 = 0.4 or 0.5 and r = 0.2. Sample 
dependence occurs. When 5 = 0.4, the state is almost wrapped around the memory state, 
and when 5 = 0.5, it moves around the mixture state in several samples. Correlated noise can 
cause a transition from the memory to the mixture state stochastically. 

Next, to verify that the state arrives at the mixture state, we consider a variation of the 
model that eliminates correlated noise after time t = 50. Figure 9 shows time evolution of the 
overlap, where 5 = 0.5 for t < 50 and 5 = for t > 50. We find that the state wandering 
around the mixture state converges on the mixture state. Since the overlaps are distributed, 
we evaluate the marginal PDF of the overlap m\ and compare the theoretical results with 
those obtained using computer simulations. Figure 10 shows the distribution of the overlap at 
time t = 51 and 60. Abscissas denote the overlap mj, and ordinates denote the marginal PDF 
p (ml) on a logarithmic scale. Solid lines denote the marginal PDF obtained theoretically, and 
boxes denote histograms obtained using computer simulations. They show positive agreement. 
We find that the states are wandering around either the memory or mixture state. 

4. conclusion 

To analyze the functional roles of correlated noise, we investigated the associative memory 
models to which independent and correlated noises are introduced. In this paper, we analyzed 
two types of associative memory models: one with auto- and weak cross-correlation connections 
and one with hierarchically correlated patterns. In the former, we showed that correlated noise 
can switch from autoassociative to sequential associative memory. Switching memory systems 
is important for information processing. The most important point is that correlated noise is 
not random, but is added to all neurons mutually. In the latter, both the memory states and 
their mixture states are stable. Using only independent noise, the stochastic transition from 
the memory to the mixture state cannot be invoked, but sample dependence does appear and 
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the stochastic transition can be invoked using correlated noise. 

In the case where sample dependence appears, the distribution of the overlaps needed to be 
analyzed to describe the macroscopic state. We, therefore, derived the macroscopic dynamic 
description as a recurrence relation form of probability density functions. The distributions 
obtained theoretically agree with those obtained using computer simulations. A transition be- 
tween attractors cannot be invoked by thermal independent noise, but can be by synchronous 
spikes, indicated by Aoyagi and Aoki. 6 ' 7 This was demonstrated by our correlated external 
input, i.e., correlated noises. In this paper, we examined the cases when p = 3 for the former, 
and p = 1 and k = 3 for the latter. When p ~ 0(1) and k ~ 0(1), the behaviors for different 
values of p and k are qualitatively same as at present excepting stability of attractors. 
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Fig. 5. Distribution of overlaps, mj, m|, and at time t — 10, 50, where e = 0.1, A = 0.1, and 
S = 0.37. Abscissas denote overlap and ordinates denote marginal PDF on a logarithmic scale. 
Boxes denote histograms obtained using computer simulations, and lines denote theoretical results, 
(a), (c), and (e) show results at t = 10, and (b), (d), and (f) show results at t = 50. 
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Fig. 6. Time evolutions of overlaps by particular common external input. Thick vertical lines denote 
the common external input. 
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Fig. 7. Retrieval processes on (m* , m\ )-planc, where r = 0.5 and (a) A = 0.1, (b) A = 0.2, (c) A 
0.3, and (d) A = 0.4. 
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Fig. 8. Time evolution of overlap with correlated noise for (a) 5 = 0.4, and (b) S — 0.5, where r = 0.2 
and A = 0.2. The results are obtained from 20 samples using computer simulations. 
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